Method and System for Detecting Network Upgrades

ABSTRACT

A system and method identify a network upgrade from a data set including a plurality of configuration sessions. The system performs the method by receiving a plurality of configuration sessions. Each of the configuration sessions comprises a plurality of configuration commands. The configuration commands are generated by a same user identifier and within a time threshold. The method further includes identifying one of the configuration sessions as a network upgrade session. The identification is based on a rareness of the configuration session or a skewness of the configuration session.

BACKGROUND

Networks continue to develop to support new functions, improvereliability and performance, and decrease operating costs. In order tosupport such development, the hardware, software, and configuration ofsuch networks may periodically be upgraded.

SUMMARY

The exemplary embodiments describe a method for identifying a networkupgrade. The method includes receiving a plurality of configurationsessions, each of the configuration sessions comprising a plurality ofconfiguration commands generated by a same user identifier and within atime threshold and identifying one of the configuration sessions as anetwork upgrade session, the identifying based on a rareness of theconfiguration session or a skewness of the configuration session.

The exemplary embodiments further describe a system for identifyingnetwork upgrades for network devices such as routers. The systemincludes a network recording device storing a data set including aplurality of configuration sessions for a plurality of network devices,each of the configuration sessions comprising a plurality ofconfiguration commands generated by a same user identifier and within atime threshold. The system further includes an analysis device receivingthe data set from the network recording device and identifying one ofthe configuration sessions as a network upgrade session, the identifyingbased on a rareness of the configuration session or a skewness of theconfiguration session.

The exemplary embodiments also describe a system comprising anon-transitory storage medium storing a set of instructions executableby a processor, to perform a method of identifying a network upgrade.The set of instructions are operable to receive a plurality ofconfiguration sessions, each of the configuration sessions comprising aplurality of configuration commands generated by a same user identifierand within a time threshold and identify one of the configurationsessions as a network upgrade session, the identifying based on arareness of the configuration session or a skewness of the configurationsession.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary system.

FIG. 2 shows an exemplary method for identifying network upgradeactivities in a system such as the exemplary system of FIG. 1.

DETAILED DESCRIPTION

The exemplary embodiments may be further understood with reference tothe following description and the appended drawings, wherein likeelements are referred to with the same reference numerals. The exemplaryembodiments describe methods and systems for detecting network upgradesthat may impact overall network performance.

Network providers may continually upgrade networks in order to supportnew functions and applications, improve performance and reliability, anddecrease operational costs. These upgrades may be planned upgrades orthey may be reactive upgrades that are the result of external networkconditions, for example, Denial of Service (DoS) attacks or wormpropagations. Possible upgrades may include the implementation ofhardware, firmware or software, the introduction of new network orservice features, and changes to network and/or device configuration.Typically, the changes are made one network element at a time (e.g., ata single router) and then applied across multiple network elements.While changes such as, for example, the introduction of new softwaregoverning the operation of a router may be tested in a controlledenvironment prior to implementation, such testing may not necessarily beable to accurately predict all possible results of implementation into alarger operational network environment. Thus, when such upgrades occur,it is important for overall network performance to be monitored in orderto detect any unintended changes or results.

However, in large-scale networks, monitoring may occur at a large numberof points, and performance changes may occur due to a large number ofchanges in network configuration being implemented by networkmaintenance personnel or automated network updating. Therefore, it isimportant to be able to identify network configuration activity, anddistinguish upgrade activity from routine maintenance. To this end, theexemplary embodiments may monitor commands send by network maintenancepersonnel as well as status at various routers within a network,identify commands that relate to changes in router configuration, andextract, from those commands, configuration sessions possessingattributes characteristic of network upgrade activities.

FIG. 1 illustrates an exemplary system 100. The system includes aplurality of routers 102, 104, 106, 108 and 110 that are connected toone another in a network 101. The network 101 includes a plurality oflinks 120 joining the routers to one another, but because the number andarrangement of these links 120 is extraneous to the scope of theexemplary embodiments, they are not identified individually in thisdisclosure. The network 101 may be a computing network, a communicationsnetwork, etc. Those of skill in the art will understand that the numberand arrangement of the routers is only exemplary and that an unlimitednumber of variations may be possible in other embodiments. The system100 may also include a control point 130, at which a user may issuecommands to the routers 102-110, performance measurements for therouters 102-110 may be received, etc. Those of skill in the art willunderstand that each of the routers 102-110 may include memory storingfirmware and software, a processor executing the firmware and software,one or more network interfaces, etc; for clarity, these elements are notshown in FIG. 1. Those skilled in the art should also understand that anactual network may have many more types of components in addition torouters (e.g., network servers, network appliances, memory devices,etc.). However, for the purposes of describing the exemplaryembodiments, the network 101 is simplified to include only the routers102-110 and the control point 130.

FIG. 2 illustrates an exemplary method 200 for identifying significantnetwork upgrades that may potentially have an impact on overall networkperformance from among a broader set of network configuration changesthat may also include routine maintenance, replacement of failedcomponents, provisioning of new customers, etc. The exemplary method 200may accomplish such identification based on measures of rareness andskewness of such activities.

In step 210, input data is received. This may include routerconfiguration and router diagnostic information “snapshots” from each ofthe routers in the system 100 for a period of time to be analyzed. Thismay be a predetermined interval of time (e.g., six months, one year,etc.), may be user-configurable, or may include all historical snapshotsfor each of the routers. A router configuration snapshot may includerouter OS version and various other configuration information such asrouting protocols, class of service, quality of service, etc. at thetime of the snapshot. A diagnostic information snapshot may includefirmware versions of each line card, hardware type, memory size, modulestatus, etc. at the time of the snapshot. Typically, both types ofsnapshots may be taken from all routers within a network on a dailybasis. Input data may also include workflow logs from a router controlinterface, such as at the control point 130. These logs may comprisecommands issued to routers, and may include the commands themselves,associated authentication procedures, network operators' usernames andterminal information, timestamps, etc.

Those skilled in the art will understand that the above data collectionis exemplary for the routers 102-110 of system 100. There may be othertypes of data that may be collected from the routers 102-110 and used ina manner consistent with the examples described below. In addition,other types of data may be collected from different types of networkequipment such as switches, servers, network devices, etc. Again, thisdata collected from other network equipment may also be used in a mannerconsistent with the examples described below.

In step 220, for each router, sequential snapshots are compared to oneanother to determine if and when router operating systems and firmwarehave been changed. Because router snapshots may be created on a dailybasis, configuration change information for routers may be obtained witha time precision of one day. However, those skilled in the art willrecognize that other granularities of router snapshots may be used thatare greater or less than one day.

Thus, at this step in the process, configuration changes at the routers102-110 have been identified. As described above, in a large networkthere may be many configuration changes on a daily basis and only someof these configuration changes are related to actual network upgrades.For example, individual customer provisioning is a continual andfrequent activity that may account for a substantial number ofconfiguration changes. The configuration changes related to customerprovisioning are unrelated to a network upgrade and are thereforeirrelevant to the analysis for the exemplary embodiments. Therefore, theexemplary embodiments present a method of filtering these and othertypes of unrelated configuration changes to result in only thoseconfiguration changes that are designed to impact how the network 101operates.

Accordingly, in step 230, workflow logs are collected from the routers102-110 and/or a system that monitors access to control sessions for therouters 102-110. The workflow logs include an entry for each commandissued to the routers 102-110 and may also include a timestamp, thenetwork operator that issued the command and the terminal from which thecommand was issued. In step 230, these workflow log entries are used tofilter out the router configuration changes that are not related toupgrades. This may be accomplished by comparing the workflow log entriesto the configuration changes determined in step 220. If a workflow lineor its negation (e.g., in some operating systems, preceding aconfiguration with “no” removes the effect of an existing configuration)does not appear in one of the router snapshots, then it may be assumednot to have lasting impact on the router, and can be safely filteredfrom consideration.

When step 230 is completed, some of the configuration changes that arenot network upgrades have been filtered, but some configurations changesthat are not upgrades (e.g., provisioning changes) may still be in theset of configuration changes. Thus, further filtering needs to beperformed to identify those configuration changes that are related tonetwork upgrades.

In step 240, the remaining workflow log entries are grouped intoconfiguration sessions. This is necessary because the intention of aconfiguration change can typically be inferred only through examining aseries of configuration commands, whereas the workflow logs themselveshave a flat structure with each command being a separate entry. Thus, itmay be necessary to group log entries into configuration sessions, whichgroup individual commands into a higher level semantic. Typically,network operations personnel and automatic configuration managementtools may perform one task at a time, and leave a sufficient gap inbetween tasks in order to allow proper verification of the effect of thechanges that have been made. Based on this, each group of workflow logentries to be grouped into one configuration session is defined as thegroup of entries that are from the same user ID and within a timethreshold of 10 minutes (e.g., each command is within 10 minutes of theimmediate prior command). Based on experience with individual networkoperations, other time thresholds may be set.

Once this has been accomplished, it may be necessary to separate outsignificant network configuration changes or upgrades from lower-leveltasks such as maintenance or provisioning of new customers. It may bedesirable to perform this task automatically and with minimum dependencyon domain expert input, which may be unreliable and easily outdated bynew router operating system versions or features. Thus, the exemplarymethod 200 looks for configuration sessions that are “out of theordinary” in some manner, as will be described below.

In steps 250, 260 and 270, the exemplary method 200 identifiesconfiguration sessions that are of a type that occurs rarely. It may beapparent to those of skill in the art that if a configuration command isused rarely over a long period of time (e.g., on the order of severalmonths), it is unlikely to be individual customer provisioningactivities (which are more frequent), and highly likely to relate to anetwork upgrade (e.g., activating a new network feature). Two challengesexist in order to use this concept to select configuration sessions ofsufficient rarity that they may be identified as network upgradesessions under the scope of the exemplary embodiments; these twochallenges will be addressed by steps 250 and 260.

The first challenge is that configuration sessions of the same type mayappear differently in workflow logs due to the use of differentparameters; for example, the command to set up BGP sessions at twodifferent routers would include different peer IP addresses, and wouldthus appear as different command lines in workflow logs. In order toaddress this challenge, in step 250 the entries in the workflow logs arede-parameterized by removing all IP addresses, device names, networkmasks, and all other numbers indicative of the specific target of thecommand.

The second challenge is that not all configuration commands aresupported across different router types or versions; thus, a command maybe misinterpreted as rare merely because most routers in the network donot recognize the command, such as because it is directed to an uncommontype of router. To remedy this issue, in step 260 the frequency of allcommands are normalized by the number of routers on which the commandhas ever appeared during the time window (e.g., a command that hasappeared eighteen times during the time window being considered, withall eighteen occurrences involving three routers, has occurred six timesper router during the time window.) Next, in step 270, a configurationcommand may be deemed to be rare if its frequency count is less than apredetermined rarity threshold (e.g., four occurrences of the samecommand per router in a six-month time window). All configurationsessions that include a rare command are deemed to be rare configurationsessions.

In step 280, the configuration sessions that have been identified asrare may be further filtered by testing them for skewness, which is acommon structural difference between provisioning configurations andupgrade-related configurations. Individual customer provisioning maytypically not repeat configuration lines, whereas upgrades may involveapplying certain configuration sessions a significant number of times todifferent line cards, different customers, different protocol sessions,etc. In one example, the implementation of a new value for acarrier-delay trigger timer over a network may involve configurationchanges to many network interfaces on each router. Thus, thecorresponding configuration session may be “skewed” in terms of thefrequency count of the different commands in the session. In order toevaluate skewness, each configuration session is evaluated separately.For each configuration session, a frequency count c_(i) is computed foreach different configuration command in the session. A mean value μ anda standard deviation a are determined for all the frequency counts c_(i)within a configuration session. If any of the frequency counts c_(i) aregreater μ+6σ, then the session is identified as a skewed configurationsession. Finally, in step 290, configuration sessions that have beendeemed to be either rare or skewed are output as network upgradeconfiguration sessions.

Those skilled in the art will understand that the above-describedexemplary embodiments may be implemented in any number of manners,including, as a separate software module, as a combination of hardwareand software, etc. For example, the method 200 may be a programcontaining lines of code that, when compiled, may be executed by aprocessor.

Those skilled in the art will understand that the above-describedexemplary embodiments may be implemented in any number of manners,including, as a separate software module, as a combination of hardwareand software, etc. For example, the method 200 may be a programcontaining lines of code that are stored on a non-transitory computerreadable storage medium that, when compiled, may be executed by aprocessor. The computing device, which may be referred to as an analysisdevice, that includes such lines of code may be connected to the network101, for example, the control point 130 may include the functionalitydescribed above or it may be a separate device that receives the datafrom the network 101 and performs the functionality described herein.

It will be apparent to those skilled in the art that variousmodifications may be made to the exemplary embodiments, withoutdeparting from their spirit or scope. Thus, it is intended that thisdisclosure cover modifications and variations of the exemplaryembodiments provided they come within the scope of the appended claimsand their equivalents.

1-20. (canceled)
 21. A method, comprising: receiving a plurality ofinput data; identifying configuration changes from the input data;generating a plurality of configuration sessions; and identifying one ofthe configuration sessions as a network upgrade session, the identifyingbased on one of A) a rareness of the configuration session, and B) askewness of the configuration session.
 22. The method of claim 21,wherein the input data comprises configuration snapshots and thereceiving the plurality of input data comprises: receiving a pluralityof configuration snapshots, each of the snapshots comprising informationdescribing a configuration of a router at a corresponding time and date;extracting configuration commands from the configuration snapshots, eachof the configuration commands having been sent to a router at aspecified time and date; generating a filtered set of configurationcommands, wherein the filtered set of configuration commands isgenerated by removing, from the configuration commands, allconfiguration commands that do not correspond to a change between afirst one of the configuration snapshots and a second one of theconfiguration snapshots.
 23. The method of claim 22, wherein thefiltered set of configuration commands further comprises unrelatedconfiguration changes and configuration changes not related to upgrades.24. The method of claim 22, wherein each of the configuration sessionscomprise the configuration commands generated by a same user identifierwithin a time threshold.
 25. The method of claim 24, further comprising:generating a plurality of de-parameterized configuration sessions byremoving, from each of the configuration commands, one of an IP address,a device name, or a network mask; dividing the configuration commandsinto a plurality of types of commands; determining, for each of thetypes of commands, a number of routers that received a command of thetype during a specified time period; determining, for each of the typesof commands, a number of instances of the type of command during thespecified time period; determining, for each of the types of commands, anormalized frequency count based on the number of routers and the numberof instances; identifying a type of commands as a rare type of commandif the normalized frequency count of the type is less than a thresholdvalue; and identifying one of the configuration sessions as a rareconfiguration session if it includes a configuration command identifiedas a rare type of commands.
 26. The method of claim 25, furthercomprising: determining a count of each of the types of commands withineach of the configuration sessions; and identifying one of theconfiguration sessions as a skewed configuration session if one of thecounts is greater than a threshold count.
 27. The method of claim 26,further comprising: determining, for each of the configuration sessions,a mean of the counts and a standard deviation of the counts, wherein thethreshold count is determined based on the mean and the standarddeviation.
 28. The method of claim 21, wherein the plurality of inputdata comprises one of: a plurality of configuration snapshots, each ofthe snapshots comprising information, at the time of the snapshot,describing routing protocols, class of service, and quality of service;a plurality of diagnostic information snapshots, each of the snapshotscomprising information, at the time of the snapshot, describing firmwareversions, hardware type, memory size, and module status; or a pluralityof workflow logs, each of the workflow logs comprising commands,authentication procedures, usernames, terminal information, andtimestamps.
 29. The method of claim 21, wherein the plurality of inputdata originate from one of routers, switches, servers, or networkdevices.
 30. A non-transitory computer readable storage medium thatstores a set of instructions executable by a processor, the set ofinstructions, when executed by the processor, causing the processor toperform operations comprising: receiving a plurality of input data;identifying configuration changes from the input data; generating aplurality of configuration sessions; and identifying one of theconfiguration sessions as a network upgrade session, the identifyingbased on one of A) a rareness of the configuration session, and B) askewness of the configuration session.
 31. The computer readable storagemedium of claim 30, wherein the input data comprises configurationsnapshots and the operation of receiving the plurality of input datacomprises suboperations comprising: receiving a plurality ofconfiguration snapshots, each of the snapshots comprising informationdescribing a configuration of a router at a corresponding time and date;extracting configuration commands from the configuration snapshots, eachof the configuration commands having been sent to a router at aspecified time and date; generating a filtered set of configurationcommands, wherein the filtered set of configuration commands isgenerated by removing, from the configuration commands, allconfiguration commands that do not correspond to a change between afirst one of the configuration snapshots and a second one of theconfiguration snapshots.
 32. The computer readable storage medium ofclaim 31, wherein the filtered set of configuration commands furthercomprises unrelated configuration changes and configuration changes notrelated to upgrades.
 33. The method of claim 22, wherein each of theconfiguration sessions comprise the configuration commands generated bya same user identifier within a time threshold.
 34. The computerreadable storage medium of claim 33, wherein the operations furthercomprise: generating a plurality of de-parameterized configurationsessions by removing, from each of the configuration commands, one of anIP address, a device name, or a network mask; dividing the configurationcommands into a plurality of types of commands; determining, for each ofthe types of commands, a number of routers that received a command ofthe type during a specified time period; determining, for each of thetypes of commands, a number of instances of the type of command duringthe specified time period; determining, for each of the types ofcommands, a normalized frequency count based on the number of routersand the number of instances; identifying a type of commands as a raretype of command if the normalized frequency count of the type is lessthan a threshold value; and identifying one of the configurationsessions as a rare configuration session if it includes a configurationcommand identified as a rare type of commands.
 35. The computer readablestorage medium of claim 34, wherein the operations further comprise:determining a count of each of the types of commands within each of theconfiguration sessions; and identifying one of the configurationsessions as a skewed configuration session if one of the counts isgreater than a threshold count.
 36. The computer readable storage mediumof claim 35, wherein the operations further comprise: determining, foreach of the configuration sessions, a mean of the counts and a standarddeviation of the counts, wherein the threshold count is determined basedon the mean and the standard deviation.
 37. The computer readablestorage medium of claim 30, wherein the plurality of input datacomprises one of: a plurality of configuration snapshots, each of thesnapshots comprising information, at the time of the snapshot,describing routing protocols, class of service, and quality of service;a plurality of diagnostic information snapshots, each of the snapshotscomprising information, at the time of the snapshot, describing firmwareversions, hardware type, memory size, and module status; and a pluralityof workflow logs, each of the workflow logs comprising commands,authentication procedures, usernames, terminal information, andtimestamps.
 38. The computer readable storage medium of claim 30,wherein the plurality of input data originate from one of routers,switches, servers, or network devices.
 39. A system, comprising: anetwork recording device storing a data set including a plurality ofconfiguration sessions for a plurality of network devices; and ananalysis device receiving the data set from the network recording deviceand identifying one of the configuration sessions as a network upgradesession, the identifying based on one of A) a rareness of theconfiguration session, and B) a skewness of the configuration session.40. The system of claim 39, wherein the analysis device identifies aconfiguration session as a network upgrade session if the configurationsession is a rare configuration session and a skewed configurationsession.